Papers with Multimodal language models
Using Perspectival Words Is Harder Than Vocabulary Words for Humans —and Even More So for Multimodal Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations of multimodal language models focus on vocabulary words with relatively stable, context-independent meanings in conversation, such as object names, colors, and verbs. |
| Approach: | They compare human and multimodal language models in their use of three word types: vocabulary, possessives, and demonstratives. |
| Outcome: | The models approach human-level performance on using vocabulary, but exhibit clear deficits with possessives and even greater difficulties with demonstratives. |
PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that process both text and speech face problems in response generation latency. |
| Approach: | They propose to extend the input and output sequences of the language model to support the parallel generation of text and speech. |
| Outcome: | The proposed model improves latency while maintaining quality of response content while maintaining the quality of the response content. |